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INTERACTIVE MULTIMEDIA COMMUNICATION SYSTEM 
5 FIELD OF THE INVENTION 

The present invention relates to a method and system 
for the production, transmission and reception of multimedia 
information over a communication network and more 
particularly to the production, transmission and reception 
10 of multimedia information with enhanced interactivity over 

such a network. 



BACKGROUND OF THE INVENTION 

Multimedia information can be transmitted in a variety 
15 of applications over a network. What is meant by multimedia 

information is information that has several parameters which 
can be enhanced or diminished depending upon the importance 
of the particular information being conveyed. For example, 
in a video environment there may be foreground and 
20 background information that is to be provided to a screen. 

The screen can typically be a video screen or a computer 
screen. The most common example of that type of system is 
an interactive video system such as seen on cable television 
networks (i.e., QVC or Home Shopping Network). These types 
25 of systems allow for a viewer to look at the screen and pick 

the type of product that he/she may want based on reviewing 
the information on the screen and then calling a particular 
telephone number to purchase the product. 
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Multimedia information can also be audio information in 
which news, weather, music and like could be transmitted 
with the use of background information such as white noise, 
background music or other information to complete the 
transmission. Multimedia information can also be a 
combination of graphics, video and music in an entertainment 
form such as Karaoke. As has been above mentioned, there 
are interactive systems presently available that allow for 
individuals to access certain video, audio and graphics 
information on a network. All presently known systems have 
the problem of requiring a significant amount of bandwidth 
to allow for the quality transmission of the multimedia 
information. 

It is known that multimedia information can require a 
significant amount of bandwidth. In the case of video 
information as much as 30 MB/ sec. bandwidth is required to 
send a quality video image along a transmission path. In 
the case of audio information, as much as 10 MB/ sec. 
bandwidth is required to allow for high quality audio signal 
to be produced accurately. 

What is meant by limited bandwidth can be viewed in two 
different ways. One way is to view the total available 
bandwidth at any instant in time. Another way of deciding 
what the particular bandwidth requirement is by determining 
the bandwidth that is available over a specified time 
period. 
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The available bandwidth therefor can be limited in two 
ways; first , the overall bandwidth of the network is limited 
such as in telephone networks where the bandwidth is 
approximately ten (10) kilohertz. However in this example, 
5 the available bandwidth over a specified period of time 

could be very high because a particular telephone set is not 
utilized constantly. 

On the other hand, in looking at a large bandwidth 
network such as cable television, although the bandwidth is 

10 significant, most of the available bandwidth is utilized for 

providing the network signals. In such an example, the 
available bandwidth over a specified period of time might be 
much less than the above-mentioned telephone system. 

Telephone networks have traditionally been used to 

15 transmit data or voice information. There are telephone 

sets that can also transmit video information over telephone 
lines. For example, American Telephone & Telegraph (AT&T) 
has introduced a phone called the Videophone which can 
transmit video images across the telephone network. It is 

20 becoming readily apparent that the telephone networks can be 

used for interactive multimedia information also. However, 
as has been mentioned above|| a typical telephone network has 
a limited bandwidth to transmit certain information. 



Particularly when attempting to transmit video information 
25 which might require significant bandwidth^ the telephone 

lines are inadequate to provide quick transmission of a high 
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quality video image. 

In addition, in such systems the communication network, 
such as a telephone network itself, will not provide the 
enhancement capabilities required to provide the 
psychographic information that will enhance interactivity. 
Finally, in a typical telephone network many of the 
technologies required for such a system has not entered the 
home. For example, although there are many known video 
enhancing capabilities, audio enhancing, printing , telephone 

j^pHliliiBiiii^iM^ call processing control systems-; aftd | 

the like in commercial use, those techniques are not 
typically part of the home environment. Hence, what is 
needed is an interactive multimedia system that provides 
high quality information over a telephone network. What is 
also required is a system that will allow for the 
production, transmission, and reception of interactive 
multimedia information that includes enhanced interactivity. 
What is also needed is a device that can be utilized within 
the home that can receive and transmit enhanced multimedia 
information and control a number of other devices for useful 
purposes. 

It is important that such a device be inexpensive to 
install in the home environment. Furthermore, it is 
important that the interactive multimedia device be one that 
when installed does not require the modification of the 
existing network. 
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The present invention provides such a system and device 
to be utilized in conjunction with a communication network 
such as a telephone network or the like. 

SUMMARY OF THE INVENTION 

An interactive multimedia communication system is 
disclosed that can be utilized with a telephone network, a 
similar switched network or in combination with a broadcast 
network such as satellite or cable. 

The interactive multimedia communication system 
comprises intcractivo multimedia mastering system moans for 
receiving program materials from a program source; 
multimedia call processing system responsive to intcractivo 
multimedia mastering system program materials for providing 
information; means interactively responsive to the 
interactive multimedia mastering system moans and the 
multimedia call processing system for controlling the flow 
of multimedia information to the multimedia call processing 
system; and a plurality of intcractivo multimedia devices 
for receiving and transmitting multimedia information to and 
from tho multimedia call processing system. 

In another facet of the invention, the interactive 
multimedia device comprises a multimedia modem means for 
receiving interactive multimedia information from tho 
telephone network; memory means for receiving interactive 
multimedia information from the multimedia modem moans; 
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moano for comprcooing at least a portion of the interactive* 
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g^ii^^ information received from the 

multimedia modem moano and the memory means; moano for 
decompressing at lcaot a portion of the intcraotivc 
multimedia information; moano for tranomitting the 
uncompressed portion of the interactive multimedia 
information received from the telephone network; mixing the 
decompressed portion with the uncompressed portion and 
providing an output; moano for providing the output of the 
interactive multimedia information to a display; and means 
for interactively controlling the multimedia modem means, 
compressing moans , decompressing moans , mixing moano and 
providing moano to provide interactive multimedia 
information to the display, — In one facet of the present 
invention there arc means to control other devices to 



perform specific functions including printing and the 
control of household devices. 

Through such a system and device a system can be 
utilized with an existing telephone network to produce high 



25 



quality multimedia information. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a block diagram of a prior art interactive 
multimedia system. 

Figure 2 is a block diagram of an interactive 
multimedia system for use in a telephone network alone or in 
combination with a cable tv broadcast feed in accordance 
with the present invention. 

Figure 2A is a block diagram of a single line 
connection between a Multimedia Call Processing System 
(MCPS) and an interactive multimedia device (IMD) . 

Figure 2A' is a flow chart of the operation of the 
system of Figure 2A. 

Figure 2B is a block diagram of a dual line connection 
between a MCPS and an IMD. 

Figure 2B' is a flow chart of the operation of the 
system of Figure 2B. 

Figure 3 is a flow chart of a first embodiment of the 
optimization method which is utilized in a telephone network 
in accordance with the present invention. 

Figure 4 is a flow chart showing the operation of a 
second embodiment of an optimization method which is 
utilized in a telephone network in accordance with the 
present invention in which a matrix of compression and 
transmission algorithms are utilized. 

Figure 5 is a flow chart showing psychographic 
enhancement in accordance with the present invention. 
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Figure 5A is a flow chart of a psychographic 
enhancement system in which both subjective and objective 
criteria are utilized. 

Figure 6 is a chart showing the cooperation of a 
compression algorithm with a transmission algorithm in 
accordance with the present invention. 

Figures 7A and 7B are block representations of digital 
information of an image file and a MIDI file. 

Figure 8 is a block representation of the multimedia 
call processing system (MCPS) in accordance with the present 
invention. 

Figure 9 is a block diagram of a general embodiment of 
an interactive multimedia device (IMD) in accordance with 
the present invention. 

Figure 10 is a block diagram of a second embodiment of 
an IMD in which the IMD is connected to a printer in 
accordance with the present invention. 

Figure 11 is a block diagram of a third embodiment of 
an IMD in which the IMD produces a facsimile in accordance 
with the present invention. 

Figure 12 is a block diagram of a fourth embodiment of 
an IMD in which the IMD is connected to a facsimile machine 
in accordance with the present invention. 

Figure 13 is a representation of a remote control 
utilized in conjunction with the system architecture of the 
present invention . 
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DETAILED DESCRIPTION OF THE INVENTION 

The present invention relates to an improvement in the 
transmission and reception of multimedia information when 
utilizing existing networks. The following description is 
presented to enable one of ordinary skill in the art to make 
and use the invention and is provided in the context of a 
patent application and its requirements. Various 
modifications to the preferred embodiment will be readily 
apparent to those skilled in the art and the generic 
principles and features described herein. 

Referring now to Figure 1, what is shown in simple 
block diagram form is |f| a prior art interactive multimedia 
system 10. In such a system 10, a sender sends and receives 
information on one side of system 10 and the receiver sends 
and receives information on the other side of the system 10. 
The system 10 can be a single device such as a CD-ROM player 
or a network capable of fully interactive (2-way) multimedia 
transmission . 

Before the present invention is described in detail 
certain background information should be discussed to 
clarify the utility of the present invention in terms of 
existing multimedia systems. In a multimedia system, 
various sensory information is provided to a receiver. In 
the case of video information, this information takes the 
form of foreground and background images that display a 
particular scene. In the case of audio information, the 
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foreground and background signals are such that the 
foreground information is speech and the background 
information is music. 

Typically, multimedia systems in whatever form provide 
this information over a single transmission line. In so 
doing, the amount and quality of the multimedia information 
is severely limited by the bandwidth of the transmission 
line. 

The present invention first differentiates between 
important and less important multimedia information by 
separating the information into primary and secondary layers 
through the use of a program model to minimize the bandwidth 
limitations. In such a system each layer will have its own 
set of parameters that are important psychographically , 
however, the secondary layer will not vary as much as the 
primary layer. 

What is meant by a program model refers to 
psychographic parameters within the multimedia system, that 
is parameters that relate to an individual's sensory 
perceptions when encountering multimedia information. These 
parameters comprise a set of unique and consistent elements 
for a particular class of multimedia information. In 
accordance with the present invention, its multimedia 
information is separated into different layers in accordance 
with the program model. Therefore, by way of example, in 
the instance of video images, the foreground and background 
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information might be divided into different layers. 
Similarly, in the case of audio information, the news 
information, weather information, or the like may be one 
layer whereas the background music may be the other layer. 

In the present invention these layers will be divided 
into primary and secondary layers in accordance with the 
information's importance relative to the program model. The 
most important information is identified and enhanced to 
provide the best quality information to the receiver of the 
multimedia information. 

In the preferred embodiment, the primary layers will be 
enhanced in such a way to provide a perceived improvement in 
quality of the multimedia information presented. In one 
embodiment the secondary layers are presented that may or 
may not be enhanced. Thereby the important information or 
the primary layers that are to be transmitted can be 
identified and selectively enhanced in accordance with the 
present invention. 

In addition, the primary layers generally can be 
enhanced through critical psychographic parameters take the 
form of spatial, color, audio, and temporal variables that 
occur in the primary or secondary layers. 

In a communication system such as a telephone network 
it is important that the multimedia information that is 
produced, transmitted and received is enhanced in some 
manner. This is necessary to ensure that high fidelity, 
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high quality information is presented to the viewer. 
Therefore, it will be possible to bring a superior product 
into the home environment. It is known, for example, it has 
been possible to provide video information over the 
telephone lines. However, it has been a problem sending 
high quality video information due to the bandwidth 
requirements that are needed to provide such high quality 
video information. 

The present invention is directed towards a method and 
apparatus for enhancing the interactive multimedia 
information transmitted across a network. To more fully 
explain such a method and apparatus, refer now to Figure 2 
which is a block diagram of the overall system architecture 
40 for an enhanced interactive multimedia system. The 
overall architecture 40 would be connected to a telephone 
network or the like so as to readily access the transmission 
lines located therein. Referring now to Figure 2, the 
system architecture 40 comprises an interactive multimedia 
mastering system (IMM) 42 which receives program source 
material. The IMM system provides information from the 
telephone network to a multimedia call processing system 
(MCPS) 44 and an auxiliary system 46. The multimedia call 
processing system 44 will provide and receive information 
from interactive multimedia devices (IMDs) 52. The 
structure of the IMDs 52 will be described in detail later 
in this specification. These IMDs 52 can take on many 
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different forms dependent upon how much intelligence is 
located in an IMD 52 relative to the system architecture 40. 

Referring now to Figures 2A and 2B what aro shown io a 
single lino connection botwccn the MCPS 44 and IMDS and a 2 
lino arrangement respectively. — Referring first to the 
single lino arrangement of Figure 2A and a flow chart 2A' . 
The MCPS receives a DTMF signal or control data signal from 

the IMD and by voice mode telephone via atop A. The MCP S 

then branches to an appropriate point according to the 
program model via otcp B. — Thereafter , the MCPS switches to 
data mode via atop C. — The MCPS transmits the appropriate 
data back to the IMD via otop D the MCPS then returns to 
voice mode via step E. 

In the two lino system of Figure 2B, tho data io 
exchanged on ono lino and audio information , for example, — is 

exchanged on tho other lino via a multimedia modem. In this 

embodiment referring to tho flow chart of Figuro 2B', an 
audio command is provided to IMD which causes tho MCPS to 
roooivo a DTMF signal via stop A, — Tho MCPS then brancheo 

via stop B' . Simultaneously the data is provided to tho 

MCPS via step B" in oynchronieation. — Thereafter the MCP S 
will transmit synchronised voico and data to tho IMD via 

step C and C" . Thon tho MCP S transmits data synchronised 

with tho audio information. 
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The system architecture 40 provides program material 

10 which will enhance the interactivity of information that is 

transmitted along the telephone network. Accordingly, what 
the system architecture 40 represents, in fact, is an 
additional network which would receive information from the 
program source which would also be part of the existing 

15 telephone system. The system architecture 40 will then be 

utilized to provide for enhanced multimedia information 
through psychographic manipulations or other enhancements to 
the systems to provide for an improved interactive 
multimedia system. 

20 The system 40 as above-mentioned includes an IMM 42 

that provides much of the optimization of the program source 
material for presentation to MCPS 44. 

To more specifically describe this optimization 
technique, refer now to Figure 3 which is a block diagram of 

25 an optimization method in accordance with the present 

invention. This optimization technique has been described 
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in detail in U.S. Patent Application Number ==^=. 
liilliiliiii, entitled, "Method for the Production and 
Transmission of Enhanced Multimedia Inf ormation. 11 , having a 

filing date of — ttWi^ifclllil , 

5 assigned to the assignee in the present invention, and that 

patent application is incorporated by reference in this 
application. The following paragraphs along with the 
accompanying figures will provide the details regarding the 
optimization method and how it will be used advantageously 

10 to provide an enhanced interactive multimedia system. 

The purpose of the I MM 42 is to provide maximum 
interactivity while at the same time providing maximum 
retention of the program model. It is also important that 
there be minimum transit time for the interactivity while 

15 the information has maximum replication. Therefore, it is 

very important that the program model psychographic 
parameters be well described. For example, the spatial, 
color, temporal, audio response, material concept, 
contention perception all should be very well described and 

20 defined in the program model. 

Referring now to Figure 3 what is shown is a first 
embodiment of a flow chart for providing an enhanced 
interactive image that utilizes the principles of the 
present invention. The flow chart 100 comprises the steps 

25 of providing a program model to a separator. The separator 

102 will divide the information into primary and secondary 
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layers of interactive multimedia information. The 
separation is automatic and can be accomplished in a variety 
of ways. For example, the layers can be separated by 
production sources. In another example, separation can be 
accomplished through key coding the layers. In yet a third 
example, the layers can be spatially separated or separated 
by the various colors. Finally, layers of information could 
be separated by a filtering process. 

The primary layers are provided to the compression 
generation block 104. There are a variety of ways that the 
multimedia data can be changed or generated to use less 
bandwidth. For example, compression algorithms or their 
equivalents could be utilized to reduce the bandwidth used. 
In addition generators, such as in a tone generator system, 
could be utilized to reduce the bandwidth required. 
Finally, key coding systems could be utilized to reduce 
bandwidth use. Such systems will be discussed in more detail 
later in the specification. 

In this embodiment, the primary layer is provided to an 
encoder where the primary layer is prepared for transmission 
(block 108) . The primary layer io then decompressed (block 
110) . Thereafter the primary layer of information is 
decompressed (block 110) . The primary layer is then decoded 
and mixed with the secondary layer of data information 
(block 112) to provide an enhanced interactive multimedia 
image (motion, video or other) or data to a display. 
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Similarly , the secondary layer is compressed through 
block 114, encoded (block 116) and then transmitted to block 
118 to decompress and mix block 110. The two signals 
(primary and secondary) are then sent to display 112. 

In this embodiment, for example, in the optimization of 
video images, the primary layer can be the foreground image, 
the secondary layer can be background information. Through 
the use of this type of optimization technique multimedia 
information can be enhanced while at the same time utilizing 
significantly less bandwidth. 

To more fully understand this feature refer now to the 
following discussion. In a typical interactive multimedia 
system| the information is all sent along one layer. The 
information that can then be transmitted is limited by the 
bandwidth of that layer. 

In the prior art, the interactive multimedia 
information that could be transmitted along typical networks 
or transmission paths that are very limited because, for 
example, in the case of vido illli images the bandwidth is 
not adequate to provide a high quality image to a display. 

Hence, in the present invention, by separating the 
multimedia information into primary and secondary layers and 
thereafter compressing the more important information 
utilizing well known compression algorithms, a system is 
described that can produce enhanced interactive multimedia 
information that i| easily fee transmitted over existing 
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networks . 

To more fully describe the psychographic enhancement 
feature of the present invention refer now to Figure 4 which 
shows the various possibilities from a particular program 
model. The program model is provided to the separator 42 of 
the multimedia system. 

Psychographic enhancements are critical to the 
improvement in interactive multimedia transmission and 
roocption enhanccmonto iii^ipill^HKi^^iii^ffiM in the 
context of the present application is information that is 
not transmitted but operates on, operates in conjunction 
with, or operates as a supplement to transmitted multimedia 
information. There are three separate categories that will 
be described that relate to psychographic enhancements. 

The first category will be described as a cross 
correlation between the information that is being 
transmitted and being enhanced due to the presence of 
information that is not transmitted. Dithering of image is 
an example of this in that dithering masks artifacts of 
images that are present and that improves the image quality. 
This type of system does not remove the artifacts but 
actually just masks imperfections. A second example in the 
audio arena where secondary audio materials such as a sound 
of an ocean or the like which might mask problems in the 
audio quality of the primary sound (voice, music or the 
like) . 
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The second category is where the signal is actually 
changed without the use of any control signal; for example, 
through interpolation or some other technigue. The typical 
example of that is a graphic equalizer in which certain 
frequencies are enhanced depending on the range of the 
particular device. Another example of the second category 
is to frequency or amplitude compress a certain signal so as 
to further enhance the frequencies that are being 
transmitted. It is also known to use various filters to 
sharpen or provide certain information that will actually 
modify the signal without controlling it per se. 

Finally, the third category is using the primary and 
secondary information to drive the other generators that 
might be present within the multimedia system. This can be 
utilized to either enhance the multimedia information or 
enhance the program model. An example of this is the use of 
real-time graphics frequency spectrum displays to enhance a 
music juke box type of program model. 

As io ocon ||||f| in Figure 4 |, the primary multimedia 
information layer can be compressed to reduce the bandwidth 
utilizing well known algorithms. It ia alao oocn that 11 
^^BPI^Hi the signal can be replaced by a generator that 
is responsive to the primary/ secondary layers signals. 
Finally, a key code could be used to cause information to be 
provided from a look-up table or the like. 

Although all of the above methods provide advantages in 
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accordance with the present invention, key coding has some 
additional non-obvious advantages when utilized in the 
optimization system of the present invention. In the 
following paragraphs the use of various key coding systems 
5 will be described generally along with their attendant 

advantages. 

Typically, when looking at an interactive multimedia 
information signal there are several components of that 
information. The first component is the data or the 

10 multimedia information itself that is being conveyed. The 

second component is referred to as program model dynamics. 
That is the changes that occur in the interactive multimedia 
information due to for example, a fade that allows for a 
transition from one scene in the graphics or video image to 

15 another. Conversely, if you want to wipe away an image 

there is information associated with the multimedia data 
that would call out for that transition to change 
efficiently. 

Finally, the third category of interactive multimedia 
20 information is what will be referred to in this 

specification which will allow a particular device or system 
to go from one category to another. In a typical 
interactive multimedia information system all this 
information is required to adequately transmit such 
25 information. 

In its simplest form, a key has an identifiable code 
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which dictates the commands on the other side of the device. 
The clearest example of such a keying system would be the 
very simple dual tone multi-frequency (DTMF) signal. This 
type of signal can be used in the telecommunications area to 
provide keying for low bandwidth protocol. These keys would 
then command a code table on the side of the network to 
provide certain information about the multimedia information 
to be displayed without requiring actual transmission of the 
multimedia information. 

A more specific version of this type of key coding is 
what will be referred to in this specification as control 
information keying. What is meant by controlled information 
keying is where a key code is utilized to access particular 
types of commands which can then be used to control other 
items on the other side of the network. 

Such a table would then be utilized to access a certain 
set of multimedia information in the network. A final 
version of key codingp^||ji|Bi wil1 be called program 
branching keying| is described by each of the keys 
representing a certain branch identification. Thus in this 
type of key coding the key is cross referenced to a 
particular branch of the interactive multimedia program 
where each of the branches allows plurality of functions or 
commands to be accessed in order to replicate the program 
model. 

The important feature that is provided by all of these 
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types of keying coding arrangements is that information 
already present on the network can be utilized. Therefore, 
the processing power inherent in the network or the system 
being accessed can be utilized rather than having to have to 
provide that processing power within the optimization system 
itself. 

It is also important to develop means to improve the 
transmission quality of the multimedia information, for 
example, the information may be transmitted utilizing a 
typical transmission algorithm such a CCITT v. 32, or Bell 
202 with standard communication file data transfer 
protocols. The interactive multimedia information could 
also utilize specialized protocols that are optimized for 
the particular interactive multimedia information that is to 
be transmitted. In so doing the algorithm for the 
compression algorithm can be interactively matrixed with the 
transmission algorithm to provide the highest quality 
information with the maximum interactivity with the minimum 
transmission line. 

Referring now to Figure ■& 1, what is shown is a flow 
chart that shows the cooperation of the transmission 
algorithm with the compression algorithm to produce a high 
quality multimedia image. The flow chart comprises 
providing a program model in which the primary and secondary 
layers are separated (block 202). The primary layer is 
compressed and encoded (blocks 204 and 206) . 
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A control element (block 210) is utilized to control a 
compression matrix and a transmission matrix. These 
two matrices comprise a plurality of compression algorithm 
and transmission algorithm respectively that are 

*************** " ¥■"»"■ " ^/i'iViV"* .™*™** * * ****** ■* m* 

interactively controlled such that as the various algorithms 
are detected^ the guality of the multimedia information and 
the speed of the transmission are interactively determined. 

The quality of the information could be determined 
manually or through the use of some control circuitry in the 
IMM. It should be understood that these same matrices could 
also be used on the secondary layer. It was not shown or 
described for purposes of clarity and simplicity. 

Psychographic quality can be objective or subjective 
dependent upon the type of system employed. Referring now 
to the flow chart of Figure 5A, what is shown is the 
operation of the system when either objective or subjective 
criteria is utilized. In this embodiment, the psychographic 



is enhanced by using a specific algorithm ouch ao 
JPEG applied to a opccific layer or as an example , a wodding 
ring that io located in a carrier of oomc oort via otcp 2A. 



— — ■"^m^M^mm^mmmm 



lllill Thereafter, it is determined whether to be a 
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subjective quality standard or an objective quality via step 
4A. If the standard is an objective quality (such as 
determining the magnitude of an error or the like) then a 
comparison is made to the objective quality or size metric 
(kilobyte storage) via step 6A. Then it must be determined 
if the enhancement meets the metric via step 8A. If the 
enhancement meets the metric then the process can end. If 
the enhancement does not meet the metric then return to step 
2A, begin process modification of same algorithm. 

If on the other hand the measurement is subjective step 
4A; then it must be determined if the enhancement is 
acceptable to the user. For example, a normal viewer 
viewing a news program provides an indicator on the output 
device. If the enhancement is not acceptable then return to 
step 2A. If the enhancement is acceptable then end. 
Whether the quality measurement is subjective or objective 
then enhancement process allows for improved 

interactivity and enhanced quality of the multimedia 
information. 

Referring now to Figure 6 what is shown is a block 
representation of a matrix of compression algorithm 

lipliiiiM with transmission algorithm ^liiii^fflii that could 
be utilized in accordance with the present invention. The 
circles 302 aligned in the vertical direction are the 
compression algorithms. The rectangles 304 aligned in the 
horizontal direction are the transmission algorithms. 
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For example! the compression algorithms could be JPEG, 

******* 

a generator with MIDI, and a key for a weather map 
background. Similarly, the transmission algorithms could be 
optimized for JPEG, data compression for MIDI, or DTMF for 
5 key transmission type algorithms. To provide the highest 

quality multimedia information while at the same time 
utilizing minimum bandwidth the different algorithms can be 
selected in an interactive manner. 

Hence, a first compression algorithm could be selected 

10 along with the first transmission algorithm. The multimedia 

information is reviewed either for image or audio qualityf 
than a second compression algorithm is selected. The 

multimedia information is reviewed and if the quality is not 
acceptable|| then a second transmission algorithm is 

15 selected. The quality of the information is ^^11 reviewed. 

This process is repeated until the highest or desired 
quality multimedia information and interactivity speed are 
provided . 

The multimedia information derived from the 
20 compression/transmission algorithms can be analog or digital 

in nature. However, in a digital signal there are certain 



other features that can be taken fee advantage HI that can be 
utilized in accordance with the present invention. 

It is known that digital data information is typically 
25 sent in a file which specifies certain parameters of that 

data and the data information itselfp and within the data 
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information itself is information which may not change for a 
certain set of files. In the case of an image file, the 
header information may specify the dimensions, pixel depth, 
and certain other features of the particular image. This 
file may take up a much as twenty percent of the data file. 

Conversely, in a file such as MIDI music file which 
comprise a plurality or a series of music notes, the header 
may include instrument information, related information and 
characteristics of that particular file. In both of the 
above mentioned examples, the header information may not 
change , through ^^^^^^H^^^i the use of the 

•"■I* 1*1 "I"* ■ ""I"!"!"!* *£* 1*1*1*1*1*1*1*1*" vC "■"■"■"■"■"■"" "jvC ■■- ■ " ■ I*""-"-"-*- 

optimization method|f the amount of information may be 
significantly reduced over time. 

Hence, in the case of the image file, the header could 
be sent first with no compression or with lossless data 
compression as the secondary file because it will always 
remain the same. The date file itself can then be 
compressed down to its smaller size. 

Another method for enhancing the psychographic 
parameters is to provide some form of error detection and 
adjustment. As has been mentioned before the detection and 
adjustment can be accomplished via interpolation of the 
error. An alternative method of error corrector is through 
an error correct ion/ transmission algorithm. What is meant 
by this, is relating the transmission to the compression to 
enhance interactivity. 



JAS103CIP REDLINED 
(PACIPRED. 103 ) 92295 



-28- 





In this type of system before the file is sent|| the 
base file is compressed and then decompressed. This 
decompressed file is called an expanded compressed base 
file. The expanded compressed base file is then compared to 
5 the original base file and an error file is then developed 

(the error file being the difference between the base file 
and the expanded compressed base file) . The error file is 
compressed and sent along with the compressed version of the 
base file down the line. These files are then combined and 

10 decompressed to provide an enhanced image. 

Referring now to Figures 7A and 7B it is seen that a 
data file utilizing this technique could initially be 
separated into primary and secondary layers. The primary 
layer could be compressed using a first compression 

15 algorithm, the header could be sent first along a first 

transmission path and the compression signal could be sent 
along a second transmission path. 

Therefore, the amount of storage necessary for the file 
is significantly reduced through secondary compression 

20 techniques. This information can then be transmitted or 

stored across the network rather than having to have all 
that information stored within a particular device within 
the optimization system. 

It should be understood by one of ordinary skill in the 

25 art will recognize that the number of algorithms is not 

limited to the number shown in the figures. In addition it 
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should be recognized that the order or the selection of the 
algorithms could be changed and that would be within the 
spirit and scope of the present invention. 

The present invention has been discussed in terms of 
5 compressing the primary layer or layer and by compressing 

and transmitting that primary layer in a particular way the 
interactivity of the system is enhanced. It should be 
understood that it may be equally important to enhance 
secondary layers to produce the same effect. 

10 Therefore, it may be important to enhance the secondary 

layer, it may be important to enhance the primary layer or 
it may be important to enhance both. Therefore, the present 
invention through the use of compression and transmission 
algorithms and through the psychographic enhancement of the 

15 program model can enhance interactivity of a multimedia 

system . 

It should also be understood that the function of the 
compression and transmission algorithms can also be done 
through other means; for example, a signal generator could 

20 be used to provide the same information. That is, a signal 

generator responsive to a particular layer or layer of 
information could be utilized to provide that information or 
some level of information that is representative of that 
layer. For example, a tone generator responsive to a signal 

25 from the secondary layer to provide the tone that would be 

representative of that secondary layer. 
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Conversely, some type of graphics generator could be 
utilized to respond to that same type of signal to provide a 
certain type of graphic image in a video system. Finally, 
it should also be understood that the psychographic 
5 parameters can be adjusted by human operator or in the 

alternative can be adjusted or modified by an automatic 
means . 

The optimized multimedia information from the program 
source material is then processed by the MCPS 44 (Figure 2) 

10 to be utilized interactively with IMDs 52. Referring now to 

Figure 8 what is shown is the preferred embodiment of a MCPS 
44. The MCPS comprises a distributed computing architecture 
500. The distributed computing architecture 44H> 
includes a master node 502 that has several server nodes 504 

15 which in turn are connected to several ports 506. When 

connected via the existing networks^ such as to a switched 
telephone network|| and then on to connection to the IMD 52|| 
which has its own processing storage and computing 
structure^ the entire network can be operated as a massive 

20 distributed computing environment. This environment shares 

all dimensions of computing, storage, transmission and 
peripheral resources (printing, product ordering, mailing 
functions, etc.). This type of computing architecture would 
include dynamic port allocation and would include 

25 incremental failure characteristics to allow for robustness 

of the MCPS 44. 
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It also is very important in this system architecture 
to have an interactive multimedia device 52 which will allow 
for the receipt of high quality multimedia information from 
the system architecture. The IMD 52 can be a printer that 
5 has been attached to a video display. It can also be a 

facsimile machine which could be utilized to received 
enhanced facsimile-like information. The IMD 52 can also be 
utilized with a television monitor to provide enhanced 
audio, video and graphic information thereto. Other 
10 connections to common household devices such as a VCR and 

household heating control systems can also be utilized by 
the IMD 52. 

Figure 9 is a preferred embodiment of an interactive 
multimedia decoder (IMD) 52 A. The IMD 52A comprises several 

15 components. The telephone line is coupled to a multimedia 

modem 522. The multimedia modem is coupled to a multimedia 
memory 524 which can be an expandable dynamic random access 
memory (DRAM) 524. The multimedia modem chip 522 provides 
data to a multimedia decompressor device 526. The multimedia 

20 memory provides data to graphics/ character generator 536, 

speech generator 538 and music synthesizer 540. 

In addition, the output of the generators 536, 538 and 
540 are provided to a video control chip 532. Video control 
532 provides signals to a standard television display and 

25 receives signals from a standard television source. The 

multimedia modem 532, the multimedia memory 524, the 
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multimedia decompressor 526, the multimedia digital/audio 
control ilii, the video control 532 and music synthesizer 
544 are all ultimately controlled by an interactive control 
interface 530 which manages the operation of all of the 
above elements. The video control 528 is coupled to a 
standard telephone keypad input or for a television remote- 
type device or a special IMD remote can be utilized in a 
variety of ways which will be discussed in detail 
hereinafter. 

Personalized and demographic information (such as the 
age, race, sex and other personal characteristics) of the 
user along with the technical information associated with 
the IMD 52 (serial no., number of generators, type etc.). 
Upon connecting the IMD 52 to the MCPS 44, the IMD 52 can 
both forward this information either at the beginning of the 
session or any time afterward. The information can then be 
updated through the MCPS 44 or directly IMD 52 through 
ocloction liiiWiljil^Miiiji^Pi^ using a keypad or remote 

********* * ******* * * ***** * * * * *^^- * * » *** ******* *** *** * *** ** + + * ***************** * ■ * ■* 

1*1*1*1*** ** »*!*!*■ *I *■*■*!* ■ * I * ■ *T*I*I * ■ * ■ *I*»*»*I*I*I*»*I*I*» » ■ * ■ •■»*• ■»"•''•*•''•''•***»*•''•*•*•*» ■ »• 

control or by receiving data from the MCPS 44. 

Another mode of operation is the delivery of multimedia 
during the period when a communication network is not in 
use. In this way, large amounts of multimedia information 
can be efficiently transmitted and stored in the IMD 52 for 
IMD for later review and enhanced interactivity. 

The function of each of the different components in a 
preferred embodiment is described in a summary fashion 
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below. 

Multimedia Modem 522 

a. Responsible for all communications between standard 
phone line, optional serial port, interface to multimedia 
5 memory, multimedia decode, audio control, and processor 

control modules. 

B. Supports standards protocol for half -duplex, full 
duplex, and half -duplex high speed operation. 

C. On-chip encode/decode capability, D/A, A/D for voice, 
10 facsimile, and data functions. 

D. Dual tone multi-frequency (DTMF) detect and generation. 

E. Auto-detect voice/ facsimile/data switch for transparent 
mode transition. 

F. Incorporates controller unit with binary file transfer, 
15 facsimile, data, and voice modes, and optional proprietary 

multimedia processor control optimized protocol firmware. 

G. Firmware allows IMD to use multimedia modem to perform 
call processing function including telephone call dialing 
and connection, unattended receipt of data and fax among 

20 other functions. 

Multimedia Memory 524 

a. Nominal DRAM or VRAM for image mixing/processing, and 
auxiliary multimedia data store. 

B. Nominal ROM for resident IMD control program. 
25 C. Optional co-resident DRAM for multimedia data store and 

program/data store. 

JAS103CIP RE DL I NED 
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D. Optional non-volatile storage (extendible) . 

E. Memory control unit for VRAM/ ROM/ DRAM and non-volatile 
storage . 

Multimedia Decode 526 

a. Responsible for real-time decompression of images 
transferred to or stored in the IMD 52 with multiple 
algorithm coding. 

B. On chip inverse discrete cosine transform processor. 

C. Reverse quantizer decoder/tables. 

D. Built-in zoom, pan, chroma key, mix from compressed 
data incorporates interfaces to video data bus, multimedia 
memory, multimedia modem, video control, and microprocessor 
control sections. 

Video Control 532 

a. Responsible for all IMD 52 video mixing, enhancements, 
and display functions. 

B. Pixel processor for mix, zoom, pan, chroma key, 
transform on pixel data, transitions. 

C. Graphics processor for figures (e.g., rectangles with 
color fill) generation, sprites, text with foreign 
characters, and scrolling. 

D. Digital to analog conversion, analog to NTSC, NTSC 
video plus stereo audio to RF. 

Graphics /Character, Speech Generator, Music Synthesizer 536, 
538 and 540 
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a. Responsible for enhancing received analog/digital 
audio , music synthesis generation, and overall analog mixing 
and audio effects. 
B. Incorporates decoding burden. 
5 C. Sampled instrument synthesis from compressed MIDI 

input . 

D. Built-in micro-controller for multi-task generation. 

E. Dual analog source mix, digital audio and synthesizer 
mix, analog audio control (volume, bass, treble, balance) 

10 for output to analog left/right audio. 

Interactive Multi-Task Processor 530 

A. Responsible for multi-task execution of resident and 
downloaded IMD code for operation in conjunction or 
independently of the MCPS. 

15 b. Master/ slave microcontroller architecture for multi- 

task control of communications, multimedia memory, 
multimedia decode, digital video control, digital 
audio/ synthesis, and interface management. 

In a preferred embodiment, the IMD 52 (Figure 2) will 

20 be utilized with a television monitor to transmit and 

receive multimedia information. 

Figures 10, 11 and, 12 show different embodiments of an 
IMD 52. Figure 10 shows an IMD 52B which has an output to a 
printer 501 and a telephone handset 540. The printer 501 

25 then is utilized to print multimedia information responsive 

to the telephone handset. 
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Figure 11 shows an IMD 52C that has an output to a 
television 700 and will print facsimile-like information 
received from the system architecture 40. This so-called 
FAX IMD 52C includes a facsimile machine internal thereto. 

Figure 12 shows an IMD 52D that has an output to a 
television 53 and also has an output connected to a fax 
machine. This will allow for a viewer to view the incoming 
facsimile messages and other multimedia information and 
then print them via a fax machine. 

Yet another preferred embodiment of the present 
invention comprises a MCPS 44 which incorporates a standard 
voice response computer or interactive voice response system 
or the like to perform digital or analog voice functions of 
the MCPS 44. In addition this apparatus would also perform 
the DTMF functions for MCPS program control. To perform the 
multimedia information transmission to the IMD 52 and to 
control the; switching between voice data , fax the system 
would utilize a modification to standard voice response 
systems, interactive voice response system or a modification 
thereof. By utilizing the appropriate IMD 52 protocol 
reconnection to a standard telephone through the above 
mentioned modified systems. Accordingly, either one 
standard telephone line or a plurality of telephone lines 
operating in parallel may be utilized for transmission from 
the MCPS 44 to the IMD 52. 

Another critical feature of the IMD is to have a remote 

JAS103CIP REDLINED 
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control that will work in conjunction with the TV or other 
display or the like to provide enhanced multimedia 
information. To more fully explain this feature refer now 
to Figure 13. The remote control 900 looks much like a 
telephone keypad. It has the numerals (0-9) and symbols (*- 
#) that are part of an ordinary telephone keypad. It 
includes an enter key 902 that is typically utilized to 
change information or change channels in the case of a 
television set. 

The control 900 would also include a volume key 904 and 
a channel or memory select key 906, a connect key 908 and a 
multimedia toggle button 910. It could also include a 
credit card slot 912. This credit card slot 912 would be 
utilized by the viewer to allow for the purchase of certain 
items directly while viewing the television screen. 

Finally, the control 900 includes special effect keys 
916, for example, for allowing for the browsing of a 
multimedia directory while simultaneously displaying a 
picture in the picture of the current broadcast TV channel. 

The remote control 900 could utilize a radio frequency 
signal or audio signal to interact with the receiver and/or 
IMD 52 for the control of the IMD, control the selection of 
multimedia information, and for the control of other 
household devices. It is also known that a more 
conventional remote control could be utilized such as one 
that controls a VCR or a television and its use would be 
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within the spirit and scope of the present invention. 

Another feature of the above- identified system 
architecture 40 of Figure 2 is that the program information 
can be linked to the network to provide for enhanced 
interactivity and program quality. For example, the program 
source can follow or be synchronized with the cable or 
broadcast feed to provide for enhanced distribution of 
program information. One specific example of this to take a 
new program, like CNN broadcasting, which consists of 
several short news items. For each news there could be 
stored large archives or program material related to a 
particular story. Hence, through multimedia interaction 
with for, example, the remote control 900 the stored program 
material can be accessed. This would allow the user to 
review the material in whatever detail was desired. 

Another example, is in an advertising feed if a 
particular item was advertised, there would be the ability 
to refer to more detailed information about the product. 

Yet another example of such a system is to have one 
service linked to another. An advertisement could be linked 
to a means for ordering the particular product being 
advertised including the placement of a phone call by the 
IMD to the desired telephone number. In so doing, the 
linked system allows for additional services to be accessed. 

The important requirements for this linked feature is 
that there is a knowledge of the contents of the system 
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program requirements and there is a knowledge of where the 
user is in the program. For example, an IMD could be used 
for channel program identification in which the telephone 
call is linked to a particular channel that has been 
selected. Another example is link demographics for targeted 
interactive advertising, hence, in an advertisement for a 
diaper, for example, there would be the facility to access 
advertisements for related items such as baby powder, baby 
oil or the like through the use of this linked approach. 

It should be understood that the IMD 52 itself could be 
utilized as a call processing system. Finally, it should be 
understood that there could be multiple MCPS 42 to provide 
for very large scale call processing. 

Although the present invention has been described in 
accordance with the embodiments shown in the figures, one of 
ordinary skill in the art recognizes there could be 
variations to the embodiments and those variations would be 
within the spirit and scope of the present invention. 
Accordingly, many modifications may be made by one of 
ordinary skills in the art without departing from the spirit 
and scope of present invention, the scope of which is 
defined solely by the appended claims. 
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WHAT 18 CLAIMED IS: 

A communications network for interactive 

multimedia transmission comprising s 

interactive multimedia mastering (IMM) — system means for 
5 receiving program materials from a program source, the IMM 

system for separating the program materials into primary and 
secondary layers ; 

multimedia call processing system responsive to the 
interactive multimedia mastering system program materials 
10 for providing information; 

means interactively responsive to the interactive 
multimedia mastering system means and the multimedia call 
processing system for controlling the flow of multimedia 
information to the multimedia call processing system; and 
15 a plurality of interactive multimedia devices — (IMD) — #ea? 

receiving and transmitting multimedia information to and 
from the multimedia call processing system, 

Scott, This is material that relates to Figure 2a and 2b 
wherein the multimedia call processing system receives a 
20 control signal from at least one of the plurality of IMDs by 

voice mode, the multimedia call processing system branches 
in accordance with the program materials, the multimedia 
call processing system including means for switching to a 
data mode, — the multimedia call processing system transmits 
25 the appropriate data back to the at least one of the 

plurality of IMDs and the multimedia call processing system 
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returno to tho voioo mode. This is the end of material 
that relates to Figure 2a and 2b. 

2. The system of Claim ± |f| in which the IMM system 
provides for subjective control of the quality of the 
multimedia information • 

3. The system of Claim i f|§ in which the IMM system 
provides for objective control of the quality of the 
multimedia information through the use of a quality metric. 

4. The system of Claim ± iH in which the multimedia 
call processing system further comprises: 

means for creating multiple multimedia files for use by 
the multimedia mastering system. 

5. The system of Claim ± |i in which the interactive 
multimedia devices are coupled to a printer to print 
information from the multimedia network. 

6. The system of Claim 5 in which the interactive 
multimedia devices are coupled to a display. 

7. The system of Claim 5 in which each of the 
interactive multimedia device has the ability to store 
information related to demographics and the characteristics 
of the interactive multimedia device and thereafter forward 
the 

information at an appropriate time. 

8. The system of Claim 7 in which interactive 
multimedia device can delay transmission of information 
stored there within until the communication network is not 
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in use. 



9. 



The system of Claim 7 in which the display 



comprises a television. 



10 



15 



20 



10. The system of Claim 9 in which control of the 
transmission of program source material is accomplished 
using a standard voice response system. 

11. The system of Claim 9 in which control of the 
transmission of program source material is accomplished 
using an interactive voice response system. 

12. The system of Claim 9 in which control of the 
transmission of program source material is accomplished 
using a computer processing system. 

i3-s — Tho oyotom of Claim 10 in which a printer io 
connoctod to the IMD. 

14. The system of claim 10 in which a facsimile machine 

is connected to the IMD. 

15. The system of claim 9 in which a telephone handset 
is used to control and select the transmission and 
presentation of program source material. 

16. The system of claim 9 in which telephone keypad is 
used to control and select the transmission and presentation 
of program source material. 

17. The system of claim 9 in which a remote control is 
used to control and select the transmission and presentation 
of program source material. 

18. The system of claim 10 in which the IMD comprises a 
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music synthesizer and a multimedia decompression means for 
generating program source material. 

19. The system of claim 10 in which the IMD comprises a 
music synthesizer and a key coding means for generating 
program source material. 

20. The system of claim 10 in which the IMD comprises a 
music synthesizer and a generator means for generating 
program source material. 

21. The system of claim 10 in which utilizes switching 
between data, voice and facsimile on a single telephone line 
to transmit program source material. 

22. The system of claim 10 in which utilizes switching 
between data, voice and facsimile on multiple telephone 
lines to transmit program source material in parallel. 

23. The system of claim 10 in which certain portions 
are designated as primary program material portions and 
certain portions of the MCPS are designated as secondary 
program material portions and are used to communicate 
primary and secondary multimedia information. 

24. The system of claim 23 in which program 
transmission can occur simultaneously. 

25. The system of claim 23 in which program material 
transmission can occur at a specified period of the day. 

26. The system of claim 23 in which the program 
transmission can occur when the telephone line is not being 
utilized. 
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37-5 — An interactive multimedia dcvioo comprising i 
means for separating the multimedia into primary and 
secondary layers based upon a program model; 

memory means for receiving interactive multimedia 
5 information form the multimedia modem means; 

moans for enhancing the primary layer from the 
multimedia modem moans in accordance with the program model 
to enhance interactivity; 
Scott, This is material that relates to Figure 5a 
10 the enhancing moans further comprising moans for 

determining whether one of a first and second approach i3 to 
be followed; first moans responsive to the determining moans 
for following the first approach for providing an objective 
metric of the perceived quality of the primary layer; second 
15 means responsive to the determining means for following the 

second approach for providing a subjective indication of the 
perceived quality of the primary layer; 

This is the end of material that relates to Figure 5a. 
means responsive to the enhancing means for 
20 decompressing the primary layer of the interactive 

multimedia information; 

means for transmitting the uncompressed portion of the 
interactive multimedia information received from the 
telephone network; 
25 moans for mixing the decompressed portion with the 

uncompressed portion and providing an output; 
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mcano for providing tho output of the intoraotivo 
multimedia information to a display; and 

mcano for intoraotivcly controlling tho multimedia 
modem mcano, — compressing mcano , docompreooing mcano ; mixing 
means and providing mcano to provide interactive multimedia 
information to tho display, 

*8-5 — The device of Claim 27 in which the separating 
means comprises keying the primary and secondary laycro to 
the psychographic parameters. 

— Tho device of Claim 27 in which the separating 
means comprises separate production sources for providing 
tho primary and secondary layers. 

9-9-s — The device of Claim 27 in which the separating 
moans comprises moans for spatially separating tho primary 
and secondary layers in accordance with the poychographic 
parameters . 

31. A system for interactively providing enhanced 
information related to a communication network, the network 
including a first set of program source materials that is 
produced for a user, the network further including a second 
set of program source material that is related to the first 
set of program source material, the second set of program 
source material is stored in a separate location than the 
first set of program source material, the system comprising 

means for linking the first set of program source 
material with the second set of program source material in 
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synchronization, the first set of program source material 
being a different media type than the second set of program 
source material; 

means coupled to the linking means for controlling the 
5 quality of the first and second sets of program source 

materials lli^ 

. WAV .W.V.VV%W.-A%V.V.%W.W.W.V^^^ 




i o i^Miii^iiffli ; and 

* » 4 * 4 * 4 ' » *'»*»'*»' , » "« " T " 

means for interactively producing the second set of 
program source material; wherein the user of the system can 
produce additional information related to the first set of 
program source material. 
15 32. The system of claim 31 including means for 

simultaneously displaying the second set of program source 
material with the first set of program source material. 

33. The system of claim 31 in which the second set of 
program source materials provides more detailed information 

20 related to the first set of program source materials. 

34. The system of claim 31 in which the second set of 
program source materials provides information that is 
different from but related to the first set of program 
source materials. 

25 35. The system of claim 31 in which the interactively 

producing means interactively displays the second set of 
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program source material while the first set of program 
source material is produced. 

36. The system of claim 31 which further comprises 
means for selecting related services based upon the second 
5 set of program source material. 

3-7-5 The system of claim 31 in which controlling means 

provides for a subjective control of tho quality of 
multimedia information . 

The system of claim 31 in which tho controlling 

10 means provideo for an objective control of tho quality of 

the multimedia information through the use of a quality 
metric. 

Scott, This is material that relates to Figure 5a 

30. The sytom of claim 31 in which tho controlling 
15 moans further comprises s 

moans for determining whether ono of a first and occond 
approach is to bo followed; 

first moans responsive to tho determining moans for 
following tho first approach for providing an objective 
20 metric of tho porcoivod quality of tho first and second sot 

of program materials; and 

occond means responsive to the determining moans for 
following tho second approach for providing a subjective 
indication of tho porcoivod quality of tho first and occond 
25 sot of program materials. 
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ABSTRACT 

A system and device is provided that enhances the 
5 interactivity of multimedia information across a 

communications network or in combination with other 
networks including telecommunication and cable broadcasting. 
The system includes a plurality of multimedia call 
processing system, — an interactive call processing system, 
10 and a plurality of interactive multimedia devices (IMDo) . 

This system has the advantage of providing compression 
and/or transmission algorithms to maximise enhancement of 
the multimedia information. 
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